Skip to content

Close the three fault-injection gaps: 8 of 8 - #576

Merged
pftg merged 1 commit into
masterfrom
close-gaps
Aug 22, 2026
Merged

Close the three fault-injection gaps: 8 of 8#576
pftg merged 1 commit into
masterfrom
close-gaps

Conversation

@pftg

@pftg pftg commented Aug 22, 2026

Copy link
Copy Markdown
Member

Follow-up to the audit (#572) and the gate fixes (#574). The three defects the suite could not see are now caught, each verified by injecting the defect, watching it fail, and reverting — per the rule #574 added.

D — frozen tenure (was blind until 2027-01-01)

The rendered assertion cannot tell derived from frozen while both read "18+". Added the missing half: the stub must use derived: tenure, not a literal. The hardcoded number IS the defect class — it is how the founding year ended up wrong in eight places on 2026-08-14, each with its own copy.

Injection → the tenure stat must be \derived: tenure` so layouts/next/landing.html computes it from site.Params.foundingYear`. A second injection froze a different stat, proving the second assertion is live rather than shadowed by the first.

G — the PurgeCSS trap (the audit called it unguardable)

It is unguardable by the visual gate, which builds production and therefore purges exactly as production does. A unit test catches it: classes styled in the register's source CSS ∩ present on a rendered element − present in the fingerprinted bundle that page links. Non-empty = a silently-purged rule. Keyed on the mechanism, not on <th>.

Injection → flags rr-th-theirs. The control is what makes it believable: the same class moved onto a <td> — an element writeStats does record — goes green. It flags the purge, not merely a new class.

Costs three file reads; the unit suite already builds production.

B + the contrast hole — computed styles, not screenshots

Deliberately not per-band screenshots: those inherit the tolerance problem (font/SVG-heavy bands need a measured floor) and still return a pixel delta rather than a ratio.

  1. The "theirs" column must compute the register's own --*-muted token. Injection → computes rgb(27,28,28), not --ed-muted ... the comparison stopped comparing (#564).
  2. A WCAG walk over every visible text run on the whole page — viewport-irrelevant, so it closes gap B — compositing translucent layers, 4.5:1 text / 3:1 large / 3:1 control fills, reporting unresolvable cases instead of guessing white, and asserting it measured >50 pairs so an empty walk cannot pass.

Injections caught: the audit's above-fold defect A at 2.31:1; a below-fold defect at y=1090 that the screenshot gate reports as 0 difference; and a 1.12:1 button fill under SC 1.4.11.

This also closes the hole the audit flagged separately: defect A was previously caught only as an anonymous pixel change, indistinguishable from an intentional recolour. Nothing measured contrast until now — and this session shipped a real 3.33:1 button that only human review caught.

Worth reading in the report

The first version of the muted-column assertion ("differs from the sibling cell") passed the injection — the injected ink was a third value, so it found a difference and approved it. Two green runs on a wrong test, exposed only by breaking it. That is the case for the rule.

Gates

unit 289 runs / 6187 assertions / 0 failures in 1.42s (was 287/6143/1.33s) · critical 38 runs, 55 screenshots compared, no failures, fixtures untouched · hugo-build clean · standardrb clean.

Honest limits

The contrast walk runs at 1920×1080 only (a mobile-breakpoint-only colour is uncovered) and covers the three register pilots, not the whole site. Both are loops, not redesigns.

🤖 Generated with Claude Code

https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ

@coderabbitai

coderabbitai Bot commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 3cf6bb0e-fed8-4d77-a4e5-22bdfde02b76


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…sted

The 2026-08-22 audit planted eight defects and the suite caught three. Three of
the five misses get gates here. Every one was injected, watched fail, reverted
and watched pass - green alone proves a test runs, not that it works.

D - a frozen tenure number. The rendered assertion cannot tell `derived` from
`frozen` while both read "18+" in 2026, and saying so in the test body is
honest but guards nothing. The freeze is plainly visible one layer down, so
NextRailTest now asserts both halves: the rendered value, and that every pilot
stub carries `derived: tenure` with no stat frozen at today's derived string.
Not a config test - the hardcoded literal IS the defect class, the way the
2026-08-14 canon audit found the founding year wrong in eight places because
each kept its own copy.

G - the PurgeCSS trap the audit called unguardable. It is unguardable by the
VISUAL gate, which builds production and so purges exactly as production does,
matching its own baseline. It is trivially visible to a set diff: styled in the
register's source CSS, present on an element in the rendered HTML, absent from
the bundle the page links. next_purge_guard_test.rb is keyed on that mechanism
rather than on `th`, so the next element type Hugo stops recording is covered
without an edit, and it reads the bundle from the page's own href - stale
fingerprinted bundles sit in the same directory.

B and the contrast hole - computed styles, not per-band screenshots. The
audit's own recommendation buys four more baselines per pilot, each needing its
noise floor measured on the platform that judges it, and still returns a pixel
delta rather than a ratio - so an intentional recolour and an AA regression stay
indistinguishable. next_pilot_contrast_test.rb asserts the comparison table's
"theirs" column computes the register's own --*-muted token, then walks every
visible text run for 4.5:1 (3:1 large, SC 1.4.3) and every control fill for 3:1
against its surround (SC 1.4.11). ~1.3s for three pages, no baselines, wired
into test:critical. It closes the hole the audit flagged separately: nothing in
this suite measured contrast as a number, and a 3.33:1 black-on-ruby button
shipped this rail on human review alone.

One mistake worth keeping. The muted-column assertion was first written as
"differs from the sibling cell and from body colour" and PASSED the exact #564
injection - in that register the "ours" column IS body colour and the injected
ink is a third value. Wrong for two runs, green both times; only the injection
said so.

Injections and failures, all reverted before the next:

  D  `derived: tenure` -> `value: "18+"`
     the tenure stat must be `derived: tenure` so layouts/next/landing.html
     computes it from site.Params.foundingYear
  D  a second stat frozen at "18+" alongside the derived one
     Expected ["Years of average developer experience"] to be empty
  G  class="rr-th-theirs" on a <th> + a matching rule
     Expected ["rr-th-theirs"] to be empty
     control: the SAME class moved onto the <td> in that row goes green, so it
     flags the purge and not merely a new class
  B  .rr-td-muted -> var(--ed-ink), below the fold
     computes rgb(27, 28, 28), not --ed-muted (rgb(107, 101, 96)).
     It resolves to --ed-ink
  A  .rr-btn-primary label -> var(--ed-ink) on the accent fill
     2.31:1 (needs 4.5:1) text a.rr-btn.rr-btn-primary
  A' .rr-td-muted -> #b5b0ab, below the fold - invisible to every screenshot
     2.15:1 (needs 4.5:1) on five cells
  A" a near-paper button fill with a readable label
     1.12:1 (needs 3.0:1) fill a.rr-btn.rr-btn-primary

Each walk asserts it measured something (>50 text pairs, >0 fills, a non-empty
styled-and-rendered set) - a walk that finds nothing is empty, and empty passes.

Gates: test:unit 289 runs / 6187 assertions / 0 failures in 1.42s (287 / 6143 /
1.33s before - the purge guard reuses the build the suite already makes);
test:critical 38 runs / 0 failures, 55 screenshots compared, no failures, no
baseline rewritten; bin/hugo-build clean; standardrb clean on the new files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
@pftg
pftg merged commit 4b0ca68 into master Aug 22, 2026
5 checks passed
@pftg
pftg deleted the close-gaps branch August 22, 2026 11:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant